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DETAILED ACTION 
1. Claims 1-43 have been examined and are pending. 



Information Disclosure Statement 

2. The information disclosure statement (IDS) submitted on 05/18/2007, 02/26/2007 
01/09/2007 and 12/21/2004 are being considered by the examiner. 



Priority 

3. Acknowledgment Is made of applicant's claim for foreign priority under 35 
U.S.C. 119(a)-(d). The certified copy has been filed in parent Application No. 
10/518576 filed on 07/01/2005. 



Claim Rejections - 35 USC § 101 

35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, maciiine, manufacture, or composition of matter, 
or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

4. Claims 17 - 20, are rejected under 35 U.S.C. 101 as claimed invention is 
directed to non-statutory subject matter. 

Claims 17-20, are directed to a "computer program written in a computer 
readable format", which does not constitute statutory subject matter such as a 
process, machine, article of manufacture or composition or matter. In contrast, a 
claimed computer-readable medium having instructions is a computer element which 
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defines structural and functional interrelationships between the instructions and the 
computer to permit the instructions' functionality to be realized, and is thus statutory. 
Examiner suggests changing the claim language to incorporate a computer-readable 
medium for storing instructions of the program. Also see pages 30 and 53 of the 
Interim Guidelines for Examination of Patent applications for Patent Subject Matter 
Eligibility. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the Invention was known or used by others in this country, or patented or described In a printed 
publication in this or a foreign country, before the invention thereof by the applicant for a patent. 

5. Claims 1- 43, are rejected under 35 U.S.C. 102(a) as being anticipated by U.S. 
Patent No. 6493872 to Rangan, P. Venkat et al., (hereinafter "Rangan"). 

As to Claims 1, 13, 17 and 21, Rangan teaches content related information 
provision apparatus, method, computer program stored on media and annotation 
system that provides related information on contents consisting of reference data 
arranged in time series, characterized by comprising (as stated in col. 5, lines 53-60, 
and col. 3, lines 53-54, method and apparatus is provided which allows a 
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authoring functioning using software modules, in a video editing mode to initiate 
tracking of any image entity or entities in a video data stream which have 
timing mariners inserted into the first video data stream): 

related information storing means that stores related information on contents (as 
stated in col. 4, lines 37-39, reading the first video data stream (related 
information) stored in a first controllable dynamic buffer for inserted frame 
identifiers identifying frames from the second data stream); 

reference information storing means that stores reference information specifying 
contents to be referred to by the related information and a data reference position in 
the contents (as stated in col. 4, lines 40-42, reading frame identifiers from the 
annotation system second data stream (reference information) stored in a 
second controllable dynamic buffer); 

and information delivering means that delivers related information and/or 
reference information (as stated in col. 4, lines 65-67, and col. 5, lines 1-3, 
apparatus and methods are provided allowing data streams to be marked while 
operated in synchronization, and to then be delivered by different networks having 
different latency effects, such that the streams are not synchronous as received, but 
may be re-synchronized using the marks provided while the streams were 
synchronous). 
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As to Claims 2, 14, 18 and 22, Rangan teaches content related information 
provision apparatus, metliod, computer program stored on media and annotation 
system, according to claims 1 , 13, 17 and 21 , characterized by further comprising: 

means that specifies an installation region of a terminal apparatus to be a 
delivery destination according to the information delivering means (as stated in col. 
18, lines 1-5, and col. 17, lines 15-16, signature application module resident in 
authoring server is initiated after and coordinate tracldng and annotation has 
been performed and synchronization of Video data stream and Annotation data 
stream is achieved in the authoring server); 

and information changing means that changes contents of related information 
and/or reference information, which should be delivered, according to the installation 
region (as stated in col. 18, lines 29-32, col. 18, lines 6-10, and col. 17, lines 55-60, 
number/time mariner-generator module generates code to represent frames in 
annotation stream and also to represent time markers in video streams. Video and 
annotation streams are given frame-specific identification and marking as well as 
time-stamped so that they may later be synchronized by using inserted data 
corresponding to the frame-specific identification. The signature streams are then 
sent to their respective broadcast and/or data-transmission systems to be sent to 
an end user). 

As to Claims 3, 15, 19 and 23, Rangan teaches content related information 
provision apparatus, method, computer program stored on media and annotation 
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system, according to claims 1 , 13, 17 and 21 , characterized in that the contents refer 
to a broadcast program, and the reference information includes information 
specifying a broadcasting station, which broadcasts or has broadcasted a program, 
and information specifying a date and time when a reference part in a program is 
broadcasted or has been broadcasted (as stated in col. 6, lines 39-50, col. 3, lines 
33-38, authoring station comprises software module that may process various 
media analog or digital not limited to common formats such as Audio Video 
Interleave AVI and Moving Pictures Experts Group MPEG. Video source may 
embody any video source (contents) that may be known in the art such as a CD- 
ROM, Satellite TV, cable TV, VCR, Internet Server. Video source may provide 
prerecorded video or live broadcast video. The first data stream is a live video 
data stream and timing markers are placed by the writer in the first data stream, 
the second data stream is an annotation data stream (reference information) 
authored in synchronization with the first data stream. The annotation data 
stream includes tracking data with time stamps derived from tracking an entity in 
the first data stream (contents)). 

As to Claims 4, 5, 16, 20 and 24, Rangan teaches content related information 
provision apparatus, method, computer program stored on media and annotation 
system, according to claims 3, 14, 18 and 23, characterized in that the reference 
information specifies a broadcasting station, which broadcasts or has broadcasted a 
program, using a channel number (as stated in col. 6, lines 39-50, col. 3, lines 33-38, 



Application/Control Number: 10/518,576 Page 7 

Art Unit: 4121 

col. 9, lines 49-52 and col. 7, lines 34-41, first data stream is a live video data 
stream (broadcasted program) and frame identifiers and timing mariners are 
implemented by various methods, such as binary numbers and are placed by the 
writer in the first data stream, timing marks in some cases are at intervals of a 
number of frames by convention and are representation of value table for storing 
values. The values being collected and stored may be kept in any logical order 
such as in a data list or the like. The second data stream is an annotation data 
stream (reference information) authored in synchronization with the first data 
stream. The annotation data stream includes tracking data (reference 
information) derived from tracking an entity in the first data stream (content). 
User interaction with such an image entity during viewing of a video can be 
programmed to provide additional network-stored information about that entity to 
suitable customer premises equipment adapted to receive and display the 
information, as an overlay on the display of the dynamic video containing the 
subject image entity). 

As to Claims 6, and 27, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, characterized in 
that the information delivering means includes identification information of a site 
handling information resources related to contents in the reference information and 
delivers the identification information (as stated in col. 13, lines 45-48, col. 15, lines 
5-8, col. 16, lines 27-35, Authoring system comprises multiple authoring stations 
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(site handling information) equipped with software capable of tracking images 
through tracldng module and provide frame by frame tracking coordinates and to 
be a vehicle through which additional annotations may be provided through user 
interface, an author may set up the parameters for tracking, with reference and add 
additional annotation such as static or moving image Icons, formatted text, 
animated graphics, to a single live or pre-recorded video feed via an innovative 
synchronous architecture which allows multiple image tracking and annotation 
operations to be performed simultaneously in near real-time such that delay of the 
broadcast video to an end user is negligible. Authoring server is provided and 
adapted to combine annotation streams through frame-specific identification 
numbers, time stamps, and markings, into one annotation stream, in this way, all 
annotations performed separately may be combined and may act in unison at the 
users end). 

As to Claims 7, and 28, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, characterized in 
that the information delivering means includes Information for correcting deviation of 
a clock in a terminal apparatus to be a delivery destination in the reference 
information and delivers the information (as stated in col. 18, lines 30-33, col. 19, 
line 67, and col. 19, lines 51-54, more than one, or several, of the synchronization 
schemes may be used in any instance. A number/time marker-generator module 
generates code to represent frames in annotation stream and also to represent 
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time markers in wcfeo stream. In some instances it may be necessary for the 
servers and the user's equipment to be synchronized in time, the clocif at the 
server end is synclironized and checked with, and corresponded clocif at the 
user's end). 

As to Claims 8, and 29, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, characterized in 
that the information delivering means includes a characteristic amount of contents at 
a reference position in the contents in the reference information and delivers the 
characteristic amount (as stated in col. 16, lines 27-34, tracl<ing module is provided 
and adapted to traclt an image and provide frame by frame tracldng coordinates 
and to be a vehicle through which additional annotations may be provided through 
user interface, an author may set up the parameters for tracking such as are 
described with reference to specific position, as well as add additional annotation 
such as static or moving image icons, formatted text, animated graphics, sounds and 
the like). 

As to Claims 9, and 30, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, characterized in 
that the information delivering means delivers plural pieces of reference information 
collectively (as stated in col. 17, lines 41-45, synchronization module is added in 
authoring server and adapted to synchronize separate (plural) annotation 
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streams (reference information) before combining them and synchronizing them 
with the output video stream). 

As to Claims 10, and 33, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, characterized in 
that the information delivering means delivers the related information and/or the 
reference information in accordance with an HTTP (Hyper Text Transfer Protocol) 
(as stated in col. 1, lines 35-38, and lines 47-55, developers are introducing 
integrated systems combining TVs with computer subsystems, so a TV may be 
used as a WEB browser, (viewing web pages in HTTP) or a PC may be used for 
enhanced TV viewingfibroadcasf programs). One may thus, with a properly 
equipped system, select to view analog TV programs, digital TV programs, 
conventional cable TV, satellite TV, pay TV from various sources, and browse the 
WWW as well, displaying WEB pages and interacting with on-screen fields and 
relational systems for jumping to related information, databases, and other WEB 
pages. The capabilities are often integrated into a single display, that is, one may 
view a broadcast presentation and also have a window on the display for WEB 
interaction). 

As to Claims 11, and 34, Rangan teaches content related information provision 
apparatus, and annotation system, according to claims 1, and 21, characterized in 
that the information delivering means delivers the related information and/or the 
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reference information in accordance with an SIVITP (Simple IVIail Transfer Protocol) 
(as stated in col. 5, lines 66-67, col. 6, lines 1-5, lines 35-38, col. 14, lines 20-22, 
original video data source may be any recorded or live broadcast source and is not 
limited as to protocol. Annotations and alterations may be made by adding 
annotations including interactive icons, text, animated graphics and sounds, to 
the input data stream source in the authoring station, and output from authoring 
station is meant to be a indication of data output, and not to indicate that there is a 
single data stream. There may be multiple streams of various protocols and is 
not limited (any protocol may be used) by the medium of transport protocols 
used for transmitting the video and the annotation data stream). 

As to Claims 12, and 35, Rangan teaches content related information provision 

apparatus, and annotation system, according to claims 1, and 21, characterized in 
that the information delivering means designates a character string, which is capable 
of identifying reference information, in a header of a delivery message at the time of 
delivery of reference information (as stated in col. 18, lines 35-40, lines 29-34, col. 
19, lines 61-65 and col. 20, lines 60-65, user interface may pre-programmed by 
an author to supply the appropriate preselected annotations in a reactive 
fashion. That is, according to a specific time interval, a signal could initiate 
annotation inserts and so on. In other embodiments, an author may physically 
enter an annotation via pressing pre-defined keys on a keyboard and so on. 
There are many known methods for inserting annotations. Several separate 
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signature metliods, for analog video, digital video, are used to insure that 
synchronization information survives to the customer's equipment end. A 
number/time marker-generator module generates code (string) to represent 
frames in annotation stream and also to represent time markers in video stream. 
Further, if the video is in a digital format, such as Motion Picture Expert's Group 
(MPEG), SMPTE-like time stamps may be inserted into the tieaders in the data 
pacl^ets which are then compared with time stamps in the control stream. 
Annotation frame numbers are written into areas associated with video frames as 
well as to the appropriate annotation frame tieaders). 

As to Claims 25-26, Rangan teaches annotation system, according to claim 21 , 
characterized in that the transmitting means transmits a name of a bulletin board, in 
which a corresponding remark is written, with the name included in the reference 
information (as stated in col. 14, lines 35-40, and col. 16, lines 51-60, FIG. 8 is a 
block diagram illustrating a multiple authoring station architecture for authoring 
system that are adapted for tracking image entities and providing additional 
annotation, add annotation such as static or moving image icons, formatted text, 
animated graphics, sounds including annotation regarding tracked entities. An 
annotation manager converts annotation data, input during annotation processes 
and the data relating to the tracked entities output from the tracking module, to 
metadata for more compact transmission in output data stream containing 
information about the various annotations added by the author of authoring 
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station and the tracking co-ordinates of the tracked entities and is analogous to the 
annotation stream (reference information). User interface provides considerable 
option and capability for entering commands to add image icons, animated graphics, 
following tracked objects or static or moving independently in the video in predefined 
manner, formatted text captions and so on). 

As to Claim 31, Rangan teaches annotation system, according to claim 21, 
characterized by further comprising means that, in writing of a remark, transmits an 
execution code for automatically acquiring reference information or urging a user to 
input a remark to a terminal apparatus side of a requiring source (as stated in col. 
14, lines 35-40, single annotated video-stream is output from display module to a 
suitable connected display monitor or screen. An input signal represents user 
interaction with an entity in video stream as it is displayed. Such a input signal 
may trigger downloading of additional detailed information regarding the 
subject of interaction. Interaction signal results from a mouse click or other input 
command such as may be initiated via a connected keyboard or a remote pointing 
device or the like). 

As to Claim 32, Rangan teaches annotation system, according to claim 21, 
characterized by further comprising means that, concerning an execution code for 
automatically acquiring reference information stored in the terminal apparatus of the 
requesting source in advance or urging a user to input a remark, transmits 
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information necessary for starting the execution code to tlie terminal apparatus (as 
stated in col. 17, lines 58-65, video/data stream signature operation is executed 
after coordinate tracking and annotation operations are performed in an authoring 
station. The signature streams are then sent to their respective broadcast and/or 
data-transmission systems to transmit information to an end user. 
Video/annotation stream capture and synctironization operations, executed via 
software on customer premises equipment is executed at the user's end for a 
single combined stream to be viewed by the user). 

As to Claim 36, Rangan teaches annotation system, according to claim 23, 
characterized by further comprising: 

means that, concerning a program series to be an object of an argument in a 
bulletin board, specifies a broadcast schedule for the next broadcast of the series 
(as stated in col. 6, lines 6-16, Purpose of the autttoring station is addition of 
innovative material as annotations and alterations to the video data stream, such 
as fexf overlay, graphic icons and logos for advertisement and interaction with 
viewer, associated with identity and address data to allow a viewer at a 
computerized end station to access advertisements and other data which may be 
associated with individual entities in the video presentation); 

and means that transmits the broadcast schedule to a terminal apparatus of a 
request source (as stated in col. 7, lines 29-40, Through tracking process and 
additional editing processes a moving region associated with the image entity in a 
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display may be made to be interactive and identifiable to an end user. User 
interaction with such an image entity during viewing of a video can be 
programmed to provide additional network-stored information about that entity to 
customer premises equipment adapted to receive and display the information. Such 
networli-stored information (broadcast schedule) is transmitted and may be 
displayed, as an overlay on the display of the dynamic video containing the subject 
image entity). 

As to Claim 37, Rangan teaches annotation system, according to claim 36, 
characterized by further comprising means that sets a bulletin board for each 
program series and performs download of the program schedule from a screen 
displaying information in the bulletin board or a screen displaying a list of remarks in 
the bulletin board (as stated in col. 22, lines 34-42, single annotated video-stream 
is output from display module to a suitable connected display monitor or screen. 
An input signal represents user interaction with an entity in video stream as it is 
displayed. Such input signal may trigger downloading of additional detailed 
information of network-stored information (broadcast schedule) regarding the 
subject of interaction. Interaction signal results from a mouse click or other input 
command such as may be initiated via a connected keyboard or a remote pointing 
device or the like). 
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As to Claim 38, Rangan teaches annotation system, according to claim 23, 
characterized by further comprising: 

means that specifies a rebroadcast schedule for a program to be an object of a 
remark (as stated in col. 23, lines 32-40, user or users that receive the video data 
via broadcast, and the annotation data via a WAN, or the Internet. Additional 
data, network-stored information (rebroadcast schedule) obtained by a user 
through interaction with a tracked entity in the video may be personalized and 
specific to the user through annotation data); 

and means that transmits the rebroadcast schedule to a terminal apparatus of a 
request source (as stated in col. 23, lines 32-40, col. 22, lines 57-67, In a case such 
as this a user would, perhaps, obtain a subscription to the service (program, 
broadcast), for other rebroadcast schedule and data delivery methods. 
Prerecorded and authored video feed from a source connected to an optional 
input module may be synchronized with a previously stored and annotated data 
stream from a source connected to a second optional input module as long as the 
signature process was applied to both streams. Interaction with tracked entities and 
the like associated with the prerecorded streams may be sent to a participating 
Internet server or the like for rebroadcast through the modem sub-module and the 
system is on-line during viewing). 

As to Claim 39, Rangan teaches annotation system, according to claim 38, 
characterized in that the transmitting means transmits the rebroadcast schedule with 
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the rebroadcast schedule included in reference information (as stated in col. 23, 
lines 32-40, user or users that receive the video data via broadcast, and the 
annotation data via a WAN, or the Internet. Additional data, networlf-stored 
information (rebroadcast sciiedule) obtained by a user through interaction with a 
tracked entity In the video may be personalized and specific to the user through 
annotation data (reference information)). 

As to Claim 40, Rangan teaches annotation system, according to claim 21, 
characterized by further comprising: 

means that designates retrieval conditions for a remark (as stated in col. 22, 
lines 34-42, single annotated video-stream is output from display module to a 
suitable connected display monitor or screen. An input signal represents user's 
interaction with an entity in video stream as it is displayed. Such a signal may 
trigger downloading of additional detailed information regarding the subject of 
interaction); 

and means that retrieves a remark across plural bulletin boards on the basis of 
the designated retrieval conditions (as stated in col. 22, lines 34-42, Interaction 
signal results from a mouse click or other input command such as may be 
initiated via a connected keyboard or a remote pointing device or the like by the 
user's). 
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As to Claim 41, Rangan teaches annotation system, according to claim 40, 
characterized in that the bulletin board system uses a keyword included in a remark 
or designated separately at the time of writing the remark as retrieval conditions (as 
stated in col. 22, lines 34-42, Interaction signal results from a mouse click or other 
input command such as may be initiated via a connected keyboard or a remote 
pointing device or the like by the user's). 

As to Claim 42, Rangan teaches annotation system, according to claim 40, 
characterized in that the bulletin board system uses a name or an ID of a user who 
has written the remark as retrieval conditions (as stated in col. 7, lines 29-40, 
Through tracking process and additional editing processes a moving region 
associated with the image entity in a display may be made to be interactive and 
identifiable to an end user. User interaction with such an image entity during 
viewing of a video can be programmed to provide additional network-stored 
information about that entity to customer premises equipment adapted to receive 
and display the information. Such network-stored information (broadcast 
schedule) is transmitted and may be displayed, as an overlay on the display of the 
dynamic video containing the subject image entity). 

As to Claim 43, Rangan teaches annotation system, according to claim 40, 
characterized in that the bulletin board system uses a date and time when the 
remark is written as retrieval conditions (as stated in col. 18, lines 29-34, 
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number/time marker-generator module generates code to represent frames in 
annotation stream and represent time mariners in video stream). 



Conclusion 

6. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

US Patent No. 7168084 to Hendricks, John S. et al., US Patent No. 7143066 to 
Shear, Victor H. et al., US Patent No. 7095450 to Holmes, Steven et al., US Patent 
No. 6173317 to Chaddha, Navin et al., are cited for reference purpose only. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Muktesh G. Gupta whose telephone number is 571- 
270-5011. The examiner can normally be reached on Monday-Friday, 8:00 a.m. - 
5:00 p.m., EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Taghi T. Arani can be reached on 571-272-3787. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273- 
8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR 
only. For more information about the PAIR system, see http://pair-direct.uspto.gov. 
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Should you have questions on access to the Private PAIR system, contact the 
Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like 
assistance from a USPTO Customer Service Representative or access to the 
automated information system, call 800-786-9199 (IN USA OR CANADA) or 571- 
272-1000. 
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/Tag hi T. Arani/ 

Supervisory Patent Examiner, Art Unit 4121 
1/25/2008 



